Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors

ABSTRACT

Systems and methods in accordance with embodiments of the invention actively align a lens stack array with an array of focal planes to construct an array camera module. In one embodiment, a method for actively aligning a lens stack array with a sensor that has a focal plane array includes: aligning the lens stack array relative to the sensor in an initial position; varying the spatial relationship between the lens stack array and the sensor; capturing images of a known target that has a region of interest using a plurality of active focal planes at different spatial relationships; scoring the images based on the extent to which the region of interest is focused in the images; selecting a spatial relationship between the lens stack array and the sensor based on a comparison of the scores; and forming an array camera subassembly based on the selected spatial relationship.

CROSS-REFERENCE TO RELATED APPLICATIONS

The current application is a continuation of U.S. application Ser. No.15/707,747, filed Sep. 18, 2017, which application is a continuation ofU.S. application Ser. No. 15/004,759, filed Jan. 22, 2016, issued onSep. 19, 2017 as U.S. Pat. No. 9,766,380 which patent is a continuationof U.S. application Ser. No. 13/782,920, filed Mar. 1, 2013, nowabandoned, which application claims priority to U.S. ProvisionalApplication No. 61/666,852, filed Jun. 30, 2012, the disclosures ofwhich are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention generally relates to actively aligning lens stackarrays with arrays of focal planes.

BACKGROUND

In response to the constraints placed upon a traditional digital camerabased upon the camera obscura, a new class of cameras that can bereferred to as array cameras has been proposed. Array cameras arecharacterized in that they include an imager array that has multiplearrays of pixels, where each pixel array is intended to define a focalplane, and each focal plane has a separate lens stack. Typically, eachfocal plane includes a plurality of rows of pixels that also forms aplurality of columns of pixels, and each focal plane is contained withina region of the imager that does not contain pixels from another focalplane. An image is typically formed on each focal plane by itsrespective lens stack. In many instances, the array camera isconstructed using an imager array that incorporates multiple focalplanes and an optic array of lens stacks.

SUMMARY OF THE INVENTION

Systems and methods in accordance with embodiments of the inventionactively align a lens stack array with an array of focal planes toconstruct an array camera module. In one embodiment, a method foractively aligning a lens stack array with a sensor that includes aplurality of focal planes, where each focal plane includes a pluralityof rows of pixels that also form a plurality of columns of pixels andeach focal plane is contained within a region of the imager array thatdoes not contain pixels from another focal plane, includes: aligning thelens stack array relative to the sensor in an initial position, wherethe lens stack array includes a plurality of lens stacks and theplurality of lens stacks forms separate optical channels for each focalplane in the sensor; varying the spatial relationship between the lensstack array and the sensor; capturing images of a known target using aplurality of active focal planes at different spatial relationshipsbetween the lens stack array and the sensor, where the known targetincludes at least one region of interest; scoring the images captured bythe plurality of active focal planes, where the resulting scores providea direct comparison of the extent to which at least one region ofinterest is focused in the images; selecting a spatial relationshipbetween the lens stack array and the sensor based upon a comparison ofthe scores of images captured by a plurality of the active focal planes;and forming an array camera subassembly in which the lens stack arrayand the sensor are fixed in the selected spatial relationship.

In another embodiment, scoring the images captured by the plurality ofactive focal planes includes computing modulation transfer function(MTF) scores for the images.

In yet another embodiment, comparison of the scores of images capturedby a plurality of the active focal planes is based upon: a comparison ofthe scores of the images captured by a plurality of the active focalplanes at the selected spatial relationship to the scores of imagescaptured by the same active focal planes at different spatialrelationships; and the variation between the scores of the imagescaptured by the active focal planes at the selected spatialrelationship.

In still another embodiment, the comparison of scores includes omittingfrom consideration an image captured by an active focal plane, when thescore of the image captured by the active focal plane fails to satisfyat least one predetermined criterion.

In a further embodiment, the at least one predetermined criterionincludes the score of the image captured by the active focal plane beingwithin a predetermined range.

In a still further embodiment, the method includes deactivating anactive focal plane, when the image captured by the active focal plane isomitted from consideration.

In yet another embodiment, the comparison of scores includes determininga mathematical relationship for each of a plurality of active focalplanes that characterizes the relationship between the scores for theimages captured by the respective active focal planes and the spatialrelationship between the lens stack array and the sensor.

In another embodiment, the comparison of scores further includescomputing a best-fit plane using the determined mathematicalrelationships, where the best-fit plane, defines a desirable spatialrelationship in accordance with predetermined criterion.

In yet another embodiment, the predetermined criterion includesmaximizing scores while minimizing the variance of the scores.

In still another embodiment: the known target includes a central regionof interest and at least one peripheral region of interest; the imagesare scored such that a score is provided for each region of interestvisible in each image, the score being indicative of the extent to whichthe respective region of interest is focused in the image; thecomparison of scores includes determining mathematical relationships foreach of a plurality of active focal planes that characterize therelationships between the scores of the extent to which the centralregion of interest is focused in the images captured by the respectiveactive focal plane and the spatial relationship between the lens stackarray and the sensor; and the scores of the extent to which the at leastone peripheral region of interest is focused in the images captured bythe respective active focal plane and the spatial relationship betweenthe lens stack array and the sensor.

In a further embodiment, the comparison of scores further includescomputing, using the determined mathematical relationships: a firstbest-fit plane that defines a spatial relationship between the lensstack array and the sensor based on each active focal plane's ability tofocus on a central region of interest according to predeterminedcriterion; a second best-fit plane that defines a spatial relationshipbetween the lens stack array and the sensor based on each active focalplane's ability to focus on the at least one peripheral region ofinterest according to predetermined criterion; and a plurality of planesincrementally spaced that lie between the first and second best-fitplanes.

In a still further embodiment, selecting a spatial relationship betweenthe lens stack array and the sensor includes using at least onepredetermined criterion to select one of: a spatial relationship definedby the first best-fit plane, a spatial relationship defined by thesecond best-fit plane, and a spatial relationship defined by one of theplurality of planes.

In a yet still further embodiment, the at least one predeterminedcriterion is based upon: at each spatial relationship defined by thecomputed planes, averaging the scores indicative of the extent to whichthe central region of interest is focused, the scores being averagedacross all active focal planes at the respective spatial relationship;at each spatial relationship defined by the computed planes, averagingthe scores indicative of the extent to which the at least one peripheralregion of interest is focused, the scores being averaged across allactive focal planes at the respective spatial relationship; andassessing the variation in the determined average scores between thespatial relationships.

In a further embodiment, aligning the lens stack array relative to thesensor in an initial position further includes: performing an initialsweep of the lens stack array relative to the sensor; capturing aninitial set of images of a known target including a central region ofinterest, at varied spatial relationships along the initial sweep, usinga plurality of active focal planes; determining focus scores for thecentral region of interest in a plurality of the captured images;determining an initial set of mathematical relationships for each of theplurality of active focal planes used to capture the initial set ofimages, where the mathematical relationships characterize therelationship between the focus scores and the spatial relationshipbetween the lens stack array and the sensor; computing an initialbest-fit plane using the initial set of mathematical relationships; andaligning the lens stack array with the computed initial best-fit plane.

In another embodiment, varying the spatial relationship between the lensstack array and the sensor involves sweeping the lens stack arrayrelative to the sensor.

In still another embodiment, the lens stack array is swept in adirection substantially normal to the surface of the sensor.

In a further embodiment, scoring the images captured by the plurality ofactive focal planes includes: determining preliminary scores for thecaptured images in accordance with a first criterion; determining scoresfor a related set of captured images in accordance with a secondcriterion; and extrapolating the preliminary scores as a function of thespatial relationship between the lens stack array and the sensor basedon the scores determined for the related set of captured images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 conceptually illustrates an array camera.

FIG. 2 illustrates an array camera module.

FIG. 3 illustrates an array camera module that employs a π filter.

FIG. 4 conceptually illustrates variations in focal length that canoccur during the manufacture of a camera module using a lens stack arrayand a sensor in accordance with embodiments of the invention.

FIG. 5 is a flowchart that illustrates a process for actively aligning alens stack array and a sensor including an array of corresponding focalplanes in accordance with an embodiment of the invention.

FIG. 6 schematically illustrates an initial configuration that may beused to actively align a lens stack array with a sensor in accordancewith an embodiment of the invention.

FIG. 7 illustrates sweeping a lens stack array with respect to a sensorin accordance with an embodiment of the invention.

FIG. 8 illustrates a target that may be used during active alignment inaccordance with many embodiments of the invention.

FIG. 9 is a flowchart that illustrates an active alignment process thatuses an iterative computation process to yield an array camera modulethat is capable of capturing and recording images that have sufficienton-axis and off-axis performance in accordance with an embodiment of theinvention.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for actively aligning alens stack array with an array of focal planes on a monolithic sensor inaccordance with embodiments of the invention are illustrated. Processesfor constructing array cameras using lens stack arrays are described inU.S. patent application Ser. No. 12/935,504, entitled “Capturing andProcessing of Images Using Monolithic Camera Array with HeterogeneousImagers”, Venkataraman et al. The disclosure of U.S. patent applicationSer. No. 12/935,504 is incorporated by reference herein in its entirety.The monolithic camera modules illustrated in U.S. patent applicationSer. No. 12/935,504 can be constructed from a lens stack array and asensor including a plurality of focal planes corresponding to theoptical channels in the lens stack array. The combination of a lensstack and its corresponding focal plane can be understood to be a‘camera module.’ Ideally, the lens stack array of an array camera isconstructed so that each optical channel has the same focal length.However, the large number of tolerances involved in the manufacture of alens stack array can result in the different optical channels havingvarying focal lengths. The combination of all the manufacturing processvariations typically results in a deviation of the actual (“firstorder”) lens parameters—such as focal length—from the nominalprescription. As a result, each optical channel can have a differentaxial optimum image location. And consequently, since the sensor ismonolithic, it typically cannot be placed a distance that correspondswith the focal length of each camera within an array camera module.Notably, these manufacturing tolerances may result in different focallengths even as between lens stack arrays fabricated from the samemanufacturing process. Thus, in many embodiments of the invention, alens stack array is actively aligned with an array of focal planes toform an array camera module that is designed to address the detrimentalimpact that the variance in focal length within a lens stack array mayhave.

In the context of the manufacture of camera systems, the term activealignment typically refers to a process for aligning an optical system(e.g. a lens stack array) with an imaging system (e.g. comprising amonolithic sensor) to achieve a final desirable spatial arrangement byevaluating the efficacy of the configuration as a function of thespatial relationship between the optical system and the imaging system.Typically, this process is implemented by using the configuration tocapture and record image data (typically of a known target) in real timeas the optical system is moving relative to the imaging system. As theoptical system is moved relative to the imaging system, the spatialrelationship between the two changes, and the characteristics of therecorded image data also change correspondingly. This recorded imagedata may then be used to align the optical system relative to theimaging system in a desired manner. For example, active alignment cangenerally be used to determine a spatial relationship that results in acamera module that is capable of recording images that exceed athreshold image quality.

A lens stack array may be actively aligned with an array of focal planesin accordance with embodiments of the invention. Importantly, activealignment in this context can be far more intricate and nuanced than itis in the context of conventional, single-lens, cameras. Foremost,because a lens stack array is typically embodied in a single integralhousing, the spatial orientation of an individual lens stack (withrespect to its corresponding focal plane) cannot be separately variedfrom that of the other lens stacks—instead, varying the spatialorientation of one lens stack invariably changes the spatial orientationof the others. Consequently, it may not be possible for multiple camerasto be spatially located at their own respective most desirablepositions. As a result, active alignment in the context of array camerasmay involve computing a final arrangement that, although does notnecessarily place each camera at its own optimal position, sufficientlyorients the lens stacks of multiple cameras so that the array cameramodule as a whole achieves a desirable level of performance.

Additionally, the active alignment of a lens stack array with an arrayof focal planes typically involves the evaluation of the efficacy ofmultiple cameras—as opposed to a single camera—at respective variedspatial relationships. In many embodiments, the efficacy of a camera isdetermined by evaluating the camera's captured and recorded images of aknown target at varied spatial relationships. For instance, theModulation Transfer Function (MTF) score—a numerical score that isindicative of a recorded image's sharpness and thus also focus—may bedetermined for a given recorded image and used to evaluate a respectivecamera at a respective spatial orientation. Moreover, the recordedimages may be evaluated at different Regions of Interest (ROIs), and inparticular at different field heights. For example, an MTF score may beassigned to each ROI within a recorded image. Thus, the correspondingcameras may be evaluated as to each ROI, and this evaluation data may beused to conclude a desirable array camera module configuration.

In several embodiments, only a subset of all of the cameras in an arraycamera module is used during the evaluation process. The cameras thatdefine this subset may be predetermined, or they may be computationallydetermined by considering an initial set of image data captured by someor all of the focal planes.

Furthermore, unlike in conventional, single-lens, cameras, the activealignment of an array camera module can involve strategically disablingcameras. For example, if the picture quality of a camera when the arraycamera module is in an estimated final arrangement is below a specifiedthreshold quality, the camera may be disabled. Thus, in many embodimentsof the invention, the active alignment process is initially used toestimate a final arrangement for an array camera module, identifycameras that should be disabled on the basis that they do not achieve athreshold quality and disabling them can improve the overall performanceof the other cameras in the camera module, and then compute a finalarrangement wherein the disabled cameras are excluded from thecomputation process. Note that this is possible because an array cameraincludes a plurality of cameras and is still operable if several of thecameras are deactivated. Namely, the array camera may be configured torely on the remaining active cameras to function, and synthesize animage based on those remaining active cameras. By allowing the arraycamera software to deactivate certain cameras, higher manufacturingyields can be achieved that can reduce the cost of the completed cameramodule.

Array cameras and systems and methods for actively aligning lens stackarrays and sensors to form camera modules for use in array cameras inaccordance with embodiments of the invention are discussed furtherbelow.

Array Camera Architectures

FIG. 1 illustrates an array camera architecture disclosed in U.S.application Ser. No. 12/935,504. The array camera 100 includes an arraycamera module 110, which is connected to an image processing pipelinemodule 120 and to a controller 130. The image processing pipeline module120 is hardware, firmware, software, or a combination for processing theimages received from the array camera module 110. The image processingpipeline module 120 is capable of processing multiple images captured bymultiple focal planes in the camera module and can produce a synthesizedhigher resolution image. In a number of embodiments, the imageprocessing pipeline module 120 provides the synthesized image data viaan output 122.

The controller 130 is hardware, software, firmware, or a combinationthereof for controlling various operational parameters of the arraycamera module 110. The controller 130 receives inputs 132 from a user orother external components and sends operation signals to control thearray camera module 110. The controller can also send information to theimage processing pipeline module 120 to assist processing of the imagescaptured by the focal planes in the array camera module 110.

Although a specific array camera architecture is illustrated in FIG. 1,camera modules constructed using active alignment processes inaccordance with embodiments of the invention can be utilized in any of avariety of array camera architectures. Camera modules that can beutilized in array cameras and processes for manufacturing camera modulesutilizing active alignment processes in accordance with embodiments ofthe invention are discussed further below.

Array Camera Modules

FIG. 2 illustrates an exploded view of an array camera module formed bycombining a lens stack array with a monolithic sensor that includes acorresponding array of focal planes as disclosed in U.S. applicationSer. No. 12/935,504. The array camera module 200 includes a lens stackarray 210 and a sensor 230 that includes an array of focal planes 240.The lens stack array 210 includes an array of lens stacks 220. Each lensstack creates an optical channel that resolves an image on the focalplanes 240 on the sensor. Each of the lens stacks may be of a differenttype. For example, the optical channels may be used to capture images atdifferent portions of the spectrum and the lens stack in each opticalchannel may be specifically optimized for the portion of the spectrumimaged by the focal plane associated with the optical channel. Morespecifically, an array camera module may be patterned with “π filtergroups.” The term π filter groups refers to a pattern of color filtersapplied to the lens stack array of a camera module and processes forpatterning array cameras with π filter groups are described in U.S.Patent Application Ser. No. 61/641,164, entitled “Camera ModulesPatterned with π Filter Groups”, Venkataraman et al. The disclosure ofU.S. Patent Application Ser. No. 61/641,164 is incorporated by referenceherein in its entirety. FIG. 3 illustrates a single π filter group,wherein 5 lenses are configured to receive green light, 2 lenses areconfigured to receive red light, and 2 lenses are configured to receiveblue light. The lens stacks may further have one or multiple separateoptical elements axially arranged with respect to each other.

A lens stack array may employ wafer level optics (WLO) technology. WLOis a technology that encompasses a number of processes, including, forexample, molding of lens arrays on glass wafers, stacking of thosewafers (including wafers having lenses replicated on either side of thesubstrate) with appropriate spacers, followed by packaging of the opticsdirectly with the imager into a monolithic integrated module.

The WLO procedure may involve, among other procedures, using adiamond-turned mold to create each plastic lens element on a glasssubstrate. More specifically, the process chain in WLO generallyincludes producing a diamond turned lens master (both on an individualand array level), then producing a negative mould for replication ofthat master (also called a stamp or tool), and then finally forming apolymer replica on a glass substrate, which has been structured withappropriate supporting optical elements, such as, for example, apertures(transparent openings in light blocking material layers), and filters.

Although the construction of lens stack arrays using specific WLOprocesses is discussed above, any of a variety of techniques can be usedto construct lens stack arrays, for instance those involving precisionglass molding, polymer injection molding or wafer level polymermonolithic lens processes. Issues related to variation in back focallength of the lens stacks within lens stack arrays are discussed below.

Back Focal Plane Alignment

An array camera module is typically intended to be constructed in such away that each focal plane (i.e. an array of pixels configured to capturean image formed on the focal plane by a corresponding lens stack) ispositioned at the focal distance of each lens stack that forms anoptical channel. However, manufacturing variations can result in thelens stack in each optical channel varying from its prescription, and inmany instances, these variations can result in each lens stack within alens stack array having a different focal length. For example,parameters that may vary amongst individual lens stacks in a lens stackarray because of manufacturing variations include, but are not limitedto: the radius of curvature in individual lenses, the conic, higherorder aspheric coefficient, refractive index, thickness of the baselayer, and/or overall lens height. As one of ordinary skill in the artwould appreciate, any number of lens prescriptions may be used tocharacterize the lens fabrication process, and the respective tolerancesmay involve departures from these prescriptions in any number of ways,each of which may impact the back focal length. Due to the monolithicnature of the sensor, the spatial relationship of the focal planes (withrespect to the lens stacks) cannot be individually customized toaccommodate this variability.

Moreover, in many instances, it is the case that a single manufacturingprocess is used to fabricate a plurality of lens stack arrays.Consequently, in addition to the aforementioned reasons, the back focallengths may further vary between lens stacks from different lens stackarrays fabricated from the same process. For instance, variability(within tolerance) in the thickness of the lens substrates and spacersemployed in the lens stack, especially those toward the sensor coverglass, may further contribute to the variability in the back focallength. Additionally, variability in the (1) thickness of the sensorcover glass, (2) the bond line thickness between the lens spacer and thesensor cover glass, and (3) any air gaps between the sensor and thecover glass, may further exacerbate the variability in the back focallengths. Thus, even with constant (nominal) process parameters duringthe lens stack array fabrication and the lens to sensor attachmentprocess, sample to sample variation may result in defocused cameramodules.

The variations in focal length that can occur in a conventional lensstack array are conceptually illustrated in FIG. 4. The array cameramodule 400 includes a lens stack array 402 in which lens stacks 404focus light on the focal planes 406 of sensor 408. As is illustrated,variance between the actually fabricated lens stack and its originalprescription can result in the lens stack having a focal length thatvaries slightly from its prescription and consequently an image distancethat does not correspond with the distance between the lens stack arrayand the sensor. Accordingly, the images formed on the focal planes ofthe sensor can be out of focus. In addition, other manufacturingtolerances associated with the assembly of the array camera moduleincluding (but not limited to) variations in spacer thickness andalignment of the lens stack array relative to the sensor can impact allof the optical channels.

Active Alignment Processes

In many embodiments, processes for actively aligning a lens stack arraywith a sensor to construct an array camera module involve reading imagedata captured by multiple focal planes on the sensor as the lens stackarray is moved relative to the sensor. The image data can be utilized toevaluate the resulting image quality at different spatial relationshipsbetween the sensor and the lens stack array and the spatial relationshipthat provides a predetermined threshold level of image quality can beutilized to construct the camera module. A process that actively alignsa lens stack array with a sensor by generally aligning the two, varyingtheir spatial relationship, evaluating the resulting configurationduring the variation, and configuring the array camera module using theevaluation data in accordance with an embodiment of the invention isillustrated in FIG. 5.

A lens stack array is generally aligned (510) with a correspondingsensor that has multiple focal planes. The combination is aligned sothat each camera within the configuration is capable of capturing andrecording images. The spatial relationship of the lens stack array withrespect to the sensor is varied (520). In several embodiments, thevariation is achieved by sweeping the lens stack array with respect tothe sensor. Sweeping can be understood to mean moving one component(i.e. either the lens stack array or the sensor) in relation to theother over time. Sweeping may be in one degree of freedom or it can beacross many degrees of freedom. As can readily be appreciated, the arraynature of the camera module means that variations in the x, y, andz-directions, and tip/tilt and rotation of the lens stack array withrespect to the sensor can all have significant impact on the imaged datacaptured by the focal planes on the sensor. Note that in many arraycameras, focus and consequently sharpness of the cameras is primarilyaffected by the z-direction and the tip/tilt of the lens stack arraywith respect to the sensor, with the tip/tilt principally affecting theperformance of the corner cameras. Conversely, in a conventional camerathat comprises only a single lens stack, the image quality of the camerais primarily driven by the optical system's ‘z-position’ with respect tothe sensor. In many embodiments, the path of the sweep is predetermined.

The quality of the captured image data is evaluated (530) at the variedspatial relationships. For example, in several embodiments of theinvention, the configuration is intermittently evaluated during a sweepof the lens stack array with respect to the sensor. In many embodiments,the configuration is evaluated by evaluating multiple cameras' capturedand recorded images of a known target at the varied spatialrelationships. In several embodiments, only a subset of theconfiguration's cameras is used for evaluation purposes. An MTF scoremay be determined for each recorded image and used to evaluate arespective camera at a respective spatial orientation. The recordedimages may also be evaluated at its different ROIs. For example, an MTFscore may be assigned to each ROI within a recorded image.

The array camera module is configured (540) using the informationobtained during evaluation. In some embodiments, the configurationinvolves concluding a spatial relationship between the lens stack arrayand the sensor that results in the corresponding array camera modulebeing able to capture and record images that exceed a threshold quality.The configuration may also involve disabling cameras that do not surpassa threshold quality. Again, because array camera modules include aplurality of cameras, they can still function even when several of thecameras are disabled. The advantage of being able to disable a camera isthat the average performance of the array including the camera may bemuch lower than the average performance of the remaining cameras whenthe disabled camera is excluded from consideration in determining theappropriate alignment of the lens stack array and sensor.

Although a process, and its variants, have been described that activelyalign a lens stack array with a corresponding array of focal planes, anyof a number of different processes may be used to actively align a lensstack array with an array of focal planes in accordance with embodimentsof the invention. An initial configuration for an active alignmentprocess in accordance with embodiments of the invention is discussedbelow.

Initial Configuration for Aligning a Lens Stack Array with an Array ofFocal Planes

Active alignment processes may begin from any number of initialconfigurations in accordance with embodiments of the invention. Aninitial configuration for an active alignment process where a devicethat is capable of orienting a lens stack array is connected to a lensstack array of a corresponding array camera module, a processor isconnected to the corresponding sensor, and a target is positioned andilluminated so that the array camera module can capture and record it inaccordance with an embodiment of the invention is illustrated in FIG. 6.The array camera module 610 includes a lens stack array 620 and a sensor630 that has corresponding focal planes. The lens stack array and thesensor are generally aligned so that they are capable of capturing andrecording images of the target 640. A device that is capable ofspatially orienting the lens stack array 640 is connected to the lensstack array 620, and a processor 660 is connected to the sensor. Thus,the processor 660 is capable of capturing and recording images from thesensor 630, while the orientation of the lens stack array 620 is beingvaried, and the active alignment process can thereby be implemented. Thecombination of the device for spatially orienting the lens stack array650 and the processor 660 can be understood to be an active alignmentmachine 670.

In many embodiments, the initial configuration involves generallyaligning the lens stack array 620 and the sensor 630 so as to ensurethat the lens stack array 620 and the sensor 630 are in sufficienttranslational and rotational alignment such that each lens stack isgenerally aligned with its corresponding focal plane. Translationalmotion here refers to motion of a system (i.e. the lens stack array 620or the sensor 630) in a direction parallel to its respective surface.Rotation here refers to rotation of a system about the Z-axis (i.e. theaxis defining the distance between the sensor and the lens stack array)relative to the other. General alignment may be achieved by, forexample, monitoring a central feature on a test chart, and moving eitherthe lens stack array or the sensor in translation (with respect to theother system) such that the central feature is centrally located withinthe central camera modules; this would indicate that the systems are insufficient translational alignment. Either system may then be rotatedwith respect to the other so that the midpoints of each lens stack arrayand its corresponding focal plane define a line that runs generallyparallel to the Z-axis. During this rotational adjustment, the systemsmay also be readjusted to preserve (or enhance) adequate translationalalignment. In this way, each lens stack array may be generally alignedwith its corresponding focal plane.

Although many embodiments of the invention employ the initialconfiguration illustrated in FIG. 6, many other embodiments employ otherinitial configurations appropriate to the requirements of specificapplications. In accordance with embodiments of the invention, anyinitial configuration may be implemented that allows the spatialrelationship between the lens stack array and the sensor to be varied,and further allows the corresponding array camera module to beevaluated, manipulated, and configured based on an evaluation of it. Thevarying of spatial relationships between the lens stack array and thesensor in accordance with embodiments of the invention is discussedbelow.

Varying the Spatial Relationship of the Lens Stack Array with Respect tothe Sensor

The spatial relationship between a lens stack array and a correspondingsensor may be varied in any number of ways. For example, an activealignment process where a lens stack array is swept in a directionsubstantially normal to the sensor's planar surface in accordance withembodiments of the invention is illustrated in FIG. 7. An array cameramodule 700 includes a lens stack array 710 and a corresponding sensor720 with an array of focal planes, and the active alignment processsweeps the lens stack array 710 in a predetermined direction 730substantially normal to the sensor's surface (the z-direction). Notethat sweeping the lens stack array in this fashion systematically variesthe focus of each camera—typically cameras will be swept in focus andthen out of focus. The array camera module may be evaluated on thevaried spatial relationships along this sweep. Active alignmentprocesses in accordance with embodiments of the invention can alsoinclude tipping, tilting, and/or rotating the lens stack array withrespect to the sensor. In many embodiments, only the distance betweenthe lens stack array and the sensor is varied in a sweep referred to asa “through focus sweep” and all relevant calculations to determine theoptimum alignment (including centering as well as focus and tip/tilt)are made from images captured during the through focus sweep using therespective curve fittings and center of gravity calculations,respectively. As can be appreciated, a through focus sweep of a skewedlens stack array already provides information about the optimum tip/tiltof the lens stack array relative to the sensor by the appropriate planefitting calculations of the peak focus positions or equalized MTF,respectively. These calculations are discussed further below.

In several embodiments, the manner in which the spatial relationshipvaries is computationally determined. For example, the manner in whichthe spatial relationship varies may be determined computationally basedupon an initial evaluation of the array camera module. Additionally, themanner in which the spatial relationship varies may change during anactive alignment process. For instance, after the lens stack array hasbeen swept in a direction substantially normal to the sensor's planarsurface, a processor may compute a different sweeping path that mayfacilitate a better configuration of the array camera module.

Although several examples have been described related to how the spatialrelationship between the lens stack array and the sensor may be varied,the spatial relationship may also be varied in any number of other waysin accordance with embodiments of the invention. The evaluation of thearray camera module at the varied spatial relationships is discussedbelow.

Evaluating the Array Camera Module

In numerous embodiments, evaluating the array camera module during theactive alignment process involves having multiple cameras capture andrecord images of a known target, and evaluating these images. The imagesmay be evaluated by assessing their focus, for example. The assessmentof the focus may be performed in any number of ways in accordance withembodiments of the invention. For example, in many embodiments, an MTFscore may be determined for a given recorded image. Generally speaking,an MTF score is an advantageous metric insofar as MTF scores amongstdifferent cameras can be directly compared with one another. In someembodiments, a recorded image may be given a ‘focus score’ which cansimilarly be used to evaluate the recorded image. For example, a focusscore may be determined by convolving a kernel over contrasting featuresin an image, where the resulting value is related to the camera'sability to focus. Unlike the MTF score, a focus score may notnecessarily be directly comparable to such scores from differentcameras; instead a focus score may be more useful in evaluating a singlecamera.

The selection of which scoring metric to use may be determined, in part,by the speed in which the scores can be calculated. For instance, if ittakes longer to compute an MTF score than to compute a focus score, thefocus score may be used in the evaluation. The selection of whichscoring metric to use may also be determined, in part, by the accuracyand precision of the score. For instance, if the MTF score is a moreprecise means for evaluating image quality, then it may be used toevaluate the camera images. Moreover, the active alignment process mayutilize several methods of evaluating a recorded image, and thesemethods may not necessarily be concurrent. For example, an evaluationbased on focus scoring may be initially used, whereas an evaluationbased on an MTF score may later be used. Additionally, the activealignment process may involve relating the different scoring metrics.For example, focus scoring may be used to evaluate the set of imagesrecorded by an array camera, and MTF scoring may be used to evaluate arepresentative subset of those images. The MTF scores for the subset maythen be normalized to the respective focus scores. And this determinedrelationship may be used to determine MTF scores for the remainingimages.

Additionally, different regions of recorded images may be evaluated,thereby providing information on a camera's quality as to specificregions. For example, in certain embodiments, images are recorded of aknown target that has multiple “Regions of Interest” (ROIs), and thecameras' recorded images of the known target are evaluated with respectto each region of interest. FIG. 8 illustrates a known target used inaccordance with many embodiments of the invention. The known target 800includes a central feature 810 that highlights a central ROI, also knownas an “on-axis” ROI. The known target further includes features 820 thathighlight “off-axis” ROIs. The target in FIG. 8 is advantageous in sofar as the edges of the features are oriented in such a way that thetangential and sagittal components of the MTF score, and thus also theastigmatism, can be directly derived and compared to prior lens testdata. Thus, many embodiments utilize the known target illustrated inFIG. 8 by evaluating the quality of each camera with respect to each ofthe five ROIs.

The target illustrated in FIG. 8 may also be used in determining a focusscore. Specifically, the determination of a focus score in conjunctionwith this target may involve convolving a kernel over areas of the imagewith contrasting features for each region of interest (e.g. thecheckerboard patterns 840 or the dark slanted square against the lightbackground 850), wherein the resulting value is proportional to thecontrast between the features. For example, the following convolutionkernel may be employed:

-   -   |−1, −1, −1, −1, −1|    -   |−1, −1, −1, −1, −1|    -   |−1, −1, 24, −1, −1|    -   |−1, −1, −1, −1, −1|    -   |−1, −1, −1, −1, −1|

This convolution kernel will yield values that are proportional to acamera's ability to resolve contrast. Note that the value will either bepositive or negative depending on whether the region being evaluated istransitioning from light to dark or dark to light. However, whether aregion of interest is transitioning from light to dark or vice versa isirrelevant to a camera's ability to focus; therefore the absolute valueof these values should be obtained. Then, a focus score for each ROI maybe obtained by averaging these absolute values for each ROI.

Although, FIG. 8 illustrates a particular known target that may be usedin accordance with embodiments of the invention, many other embodimentsutilize other known targets appropriate to the requirements of specificapplications. For instance, the off-axis ROIs may be placed in thecorners of the target—this allows the performance of the camera to betested at larger field heights. In the illustrated embodiment, the ROIshave the advantage that the edges of the features are oriented in such away that the tangential and sagittal components of the MTF and thus alsothe astigmatism can be directly derived and compared to prior lens testdata. Moreover, although specific examples of how a focus score may begenerated are provided, any of a variety of techniques can be used togenerate a focus score. More generally, the evaluation techniques hereindescribed are merely illustrative. Any techniques for evaluating theefficacy of an array camera module may be incorporated in accordancewith embodiments of the invention. Using the evaluation data toconfigure the array camera module is discussed below.

Configuring the Array Camera Module

Evaluation data may be used to configure the array camera module in anumber of respects. In many embodiments the array camera module isconfigured to minimize the detrimental impact caused by variance offocal length within a lens stack array. As described above, variancewithin a lens stack array may be caused by manufacturing processvariations including (but not limited to) those that affect thefollowing parameters: the radius of curvature in individual lenses, theconic, higher order aspheric coefficient, refractive index, thickness ofthe base layer, and/or overall lens height. Additionally, as describedabove, the following manufacturing variations related to the fabricationof multiple lens stack arrays and camera modules may further exacerbatethe variability in back focal lengths: the thickness of the lenssubstrates and spacers employed in the stack, especially those towardthe sensor cover glass, the thickness of the sensor cover glass used,bond line thickness between the lens spacer and the sensor cover glass,and any air gap between the sensor and the sensor cover glass. Thus,many embodiments evaluate the quality of each camera as a function ofits spatial relationship to the sensor; thereafter, the information isused to orient the lens stack array with respect to the sensor so thatany deterioration in the quality of the array camera due to the variancein focal length within the lens stack array is lessened.

Several embodiments generate mathematical equations that approximatelycharacterize data related to camera quality as a function of spatialrelationship, and use the derived equations to compute a desired spatialrelationship that lessens the detrimental impact of variance in focallength. For example, some embodiments generate polynomial equations thatapproximately model the focal scoring data. Note that because of thenature of optics, each lens will typically have a peak focal value, andtherefore polynomial equations are well suited to characterize the data.In many embodiments, the polynomial equations are generated bydetermining coefficients for predetermined generic polynomial equations(i.e. those with undetermined coefficients), such that the resultingequation approximately characterizes the data relating the cameraquality to the spatial relationship. Many embodiments then use thesederived equations to compute a best fit plane that characterizes aspatial relationship that reduces the detrimental impact of variance infocal length.

Notably, the best-fit planes may be computed in any number of ways. Forinstance, the best-fit plane may be computed to be a plane that includesan approximation of the peak values of the polynomial equations thatcharacterize focal scoring data as a function of the spatialrelationship. But, as described above, focal scoring data may notnecessarily be directly comparable across different cameras. Therefore,best-fit planes may also be computed by generating equivalent MTFscores, and determining a plane that maximizes the mean MTF score whileminimizing its variance. Specifically, the best-fit planes may becomputed to determine a plane wherein the MTF scores amongst thedifferent lens stacks are equalized within some specified tolerance.Moreover, any number of balancing algorithms may be employed toeffectuate this computation as appropriate to the requirements of aspecific application. The determination of these planes may then be usedto facilitate the configuration of the array camera module.

In several embodiments, the configuration process involves orienting thelens stack array with respect to the sensor to form an array cameramodule that is capable of achieving pictures that have desiredcharacteristics. In some embodiments, the lens stack array is orientedwith respect to the sensor so as to achieve an array camera module thatis capable of recording images, wherein the quality of the on-axisaspects of the recorded image exceeds a specified threshold criterion.In several embodiments, the lens stack array is actively aligned withrespect to the sensor to achieve an array camera module that is capableof recording images, wherein the quality of the off-axis aspects of therecorded image exceeds a specified threshold criterion. Note also thatin various embodiments, the configuration process may involve disablingcameras that are above a certain threshold quality so as to avoidbiasing the best fit plane determination. In numerous embodiments, thelens stack array is actively aligned with respect to the sensor toachieve an array camera module that is capable of recording images,wherein the quality of both on-axis and off-axis regions of interestexceed respective specified threshold qualities.

In many embodiments, the configuration process involves disablingcameras that perform above or below a certain defined threshold quality.Again, because an array camera module has many cameras, it is possiblefor it to maintain functionality even when some of its cameras arenon-functional. In several embodiments, cameras are disabled when theirquality, as determined by their ability to focus sharply when in a givenspatial orientation, is above or below a threshold value. For example,some embodiments determine whether a camera should be disabled byevaluating an MTF score of its respective recorded images. In manyembodiments, if the number of disabled cameras exceeds a specifiedvalue, then the array camera module is designated unacceptable. Inseveral embodiments, different threshold values can be specified fordifferent types of cameras within the array camera module. For example,in a number of embodiments that employ π filter groups, differentthreshold values can be specified for the green cameras, the redcameras, and the blue cameras.

In various embodiments, information obtained during the evaluationaspect of the active alignment process is used to configure thefunctionality of the each camera. For example, if it is determined thata particular camera has a focal length that makes it better suited torecord images of objects that are at a further distance, the arraycamera module can be configured to rely more heavily on that camera whensynthesizing recorded images of objects at further distances.

The above descriptions regarding configuring an array camera module inaccordance with embodiments of the invention is not meant to beexhaustive. Indeed, array camera modules can be configured in any numberof ways based on evaluations of the configuration in accordance withembodiments of the invention. Active alignment processes that configurearray camera modules so that they are capable of capturing and recordingimages that have desirable image properties are discussed below.

Active Alignment Processes that Yield Array Camera Modules Capable ofRecording Images that have Desirable Characteristics

Active alignment processes in accordance with embodiments of theinvention can use a variety of metrics to evaluate the image data thatis captured during the active alignment process. In several embodiments,the active alignment process can optimize image quality in specificregions of the captured images, can optimize image quality in multipleregions of interest and/or can utilize a variety of metrics including(but not limited to) focus scoring and MTF scoring. An active alignmentprocess that uses an iterative computation process to yield an arraycamera module that is capable of capturing and recording images thathave sufficient on-axis and off-axis performance in accordance with anembodiment of the invention is illustrated in FIG. 9.

The process is initially configured (902) so that a lens stack array anda corresponding sensor are mounted to an active alignment machine in amanner similar to that seen in FIG. 6, so that they are generallyoperable as an array camera. This may include generally aligning thelens stack array with its corresponding sensor, which itself may includeverifying that the lens stack array and the sensor are in sufficientrotational alignment such that each lens stack is generally aligned withits corresponding focal plane, as described above. A known target withan on-axis ROI and off-axis ROIs (similar to that depicted in FIG. 8) ispositioned and illuminated so that the array camera module may captureand record its image. The initial configuration may also includedeactivating specific cameras in a predetermined fashion so that they donot record images during the alignment process.

The lens stack array is swept (904) in a direction normal to thesensor's planar surface, in a manner similar to that seen in FIG. 7, andmay be swept for a predetermined distance. During the sweep, the activecameras intermittently capture and record (906) images of the knowntarget. The processor evaluates (908) the recorded images and assigns a‘focus score’ for each region of interest in each recorded image foreach camera. Polynomial equations are derived (910) for each region ofinterest captured by each camera that best characterizes the focus scoreas a function of the camera's distance from the sensor. In someembodiments, the polynomial equations are derived by calculatingcoefficients for a given a predetermined generic polynomial equation(i.e. a polynomial equation with undetermined coefficients). Thepolynomial equations will typically have a peak value.

An “on-axis best fit plane” is derived (912) using the peak values ofthe polynomial equations. The on-axis best fit plane, is characterizedin that it maximizes the peak values corresponding to the active camerasand/or minimizes the variance in the peak values.

The lens stack array is then aligned (914) with the computed best fiton-axis plane. Each active camera captures and records (916) an image ofthe known target. Each recorded image is then evaluated (918) bydetermining an MTF score for each ROI. Cameras that do not meet athreshold MTF score are disabled (920). For example, any cameras that donot have an MTF score within 20% of the median on-axis MTF score may bedisabled, and subsequently excluded from further alignment positioncalculations. This threshold may of course be configurable. In otherembodiments, other criteria are utilized to determine which camerasshould be disabled. Moreover, if a specified number of cameras aredisabled, the array camera is deemed unacceptable.

Assuming the camera is not deemed unacceptable, the previously acquiredfocus scoring data is scaled (922) using the peak focus score and MTFscores. For example, the MTF Score may be scaled in accordance with thefollowing formula:Scaled Focus Score_(z)=(Focus Score_(z)/Peak Focus Score)*MTF Score

where the z subscript reflects the score at a particular z-position.

The focus scoring data (absolute values) are exposure/signal-leveldependent. Thus different cameras (e.g. blue, green, red cameras) willhave different absolute focus score peak values due to their differentsignal levels. However, MTF is a metric that is invariant to signallevel. Thus, MTF enables the curves for focus score to be normalizedsuch that the curve derived from focus score can also be used to compareeach camera's peak performance and not only the position at which peakperformance occurs. In other embodiments, any of a variety of metricsappropriate to a specific application can be utilized in determiningcamera peak performance.

As before, polynomial curves may then be derived (924) that characterizethe scaled focus scores. Thus, each active camera will be characterizedby polynomial equations that characterize the camera's ability toresolve each respective region of interest. Given these new polynomialequations, a best-fit on axis plane and a best-fit off axis plane arederived (926); in this instance, the best-fit planes are characterizedin that they approximately maximize the mean MTF scores while minimizingtheir variance. A configurable number of planes that are evenly spacedbetween the two best-fit planes (on-axis and off-axis) are computed(928). Scaled focus scores for each camera at their respectivecorresponding positions along each of those planes are calculated (930).A best-fit plane determined (932) wherein any deviation toward thebest-fit off axis plane causes a gain in the off-axis scaled focus scoreand a loss in the on-axis scaled score, wherein the ratio of theoff-axis score gain to the on-axis score loss falls below a configurablethreshold. The lens stack array is then re-aligned (934) with thiscomputed plane.

The efficacy of the process is verified (936). This may be accomplishedby, for example, having each active camera record an image of the knowntarget, determining an MTF score for each ROI within that image, andensuring that each MTF score surpasses some threshold calculation.

The processes described may be iterated (938) until a desiredconfiguration is achieved.

Although a particular process, and its variants, is discussed above, anynumber of processes may be used to achieve an array camera module thatis capable of capturing and recording images that have adequate on-axisand off-axis performance in accordance with embodiments of theinvention. Moreover, although the discussed process regards adequatelybalancing on-axis and off-axis performance of an array camera module,active alignment processes can be tailored to achieve any number ofdesirable picture characteristics in accordance with embodiments of theinvention.

Although the present invention has been described in certain specificaspects, many additional modifications and variations would be apparentto those skilled in the art. It is therefore to be understood that thepresent invention may be practiced otherwise than specificallydescribed. Thus, embodiments of the present invention should beconsidered in all respects as illustrative and not restrictive.

What is claimed is:
 1. A method for disabling lens stacks within anarray of lens stacks, the method comprising: providing a lens stackarray having a plurality of focal planes in conjunction with a sensor,where each focal plane comprises a plurality of rows of pixels that alsoform a plurality of columns of pixels and each focal plane is associatedwith a lens stack of the array; determining a final spatial relationshiparrangement between the lens stack array and the sensor for use in anarray camera module; capturing at least one image of a first knowntarget utilizing the plurality of focal planes at the final spatialrelationship arrangement between the lens stack and array; for eachfocal plane of the plurality, scoring the at least one image captured,where the resulting score provides an indication to which at least oneregion of interest within the first known target is focused in the atleast one image; determining that the resulting score of a focal planedoes not meet a predetermined quality criterion; and deactivating thelens stack associated with the focal plane that does not meet apredetermined quality criterion based on the resulting score of thefocal plane not meeting the predetermined quality criterion; capturingimages of a second known target using a plurality of active focal planesat different spatial relationships between the lens stack array and thesensor, where the lens stack array comprises a plurality of lens stacks,wherein the second known target has an on-axis and an off-axis region ofinterest; scoring the images captured by the plurality of active focalplanes, where the resulting scores provide a direct comparison of theextent to which the on-axis and off-axis region of interest are eachfocused in the images; comparing the resulting scores of capturedimages, wherein the comparison of scores further comprises computing: afirst best-fit plane that defines a spatial relationship between thelens stack array and the sensor based on each active focal plane'sability to focus on the on-axis region of interest according to a firstpredetermined criterion; a second best-fit plane that defines a spatialrelationship between the lens stack array and the sensor based on eachactive focal plane's ability to focus on the off-axis region of interestaccording to a second predetermined criterion; and at least a thirdintervening plane between the first and second best-fit planes; anddetermining the final spatial relationship arrangement between the lensstack array and the sensor utilizing the comparison of the scores. 2.The method of claim 1, wherein the predetermined quality criterion isbased on a lens stack's ability to focus sharply.
 3. The method of claim1, wherein the predetermined quality criterion is based upon amodulation transfer function (MTF) score for the at least one image. 4.The method of claim 1, further comprising: when the number ofdeactivated lens stacks remains below a predetermined threshold,incorporating the lens stack array within a camera module.
 5. The methodof claim 1, wherein the first known target and the second known targetare the same known target.
 6. The method of claim 1, wherein scoring theimages captured by the plurality of active focal planes comprisescomputing modulation transfer function (MTF) scores for the images. 7.The method of claim 1, wherein the comparison of the scores of imagescaptured by a plurality of the active focal planes is based upon: acomparison of the scores of the images captured by a plurality of theactive focal planes at the selected spatial relationship to the scoresof images captured by the same active focal planes at different spatialrelationships; and the variation between the scores of the imagescaptured by the active focal planes at the selected spatialrelationship.
 8. The method of claim 1, wherein the final spatialrelationship arrangement minimizes the variance of the scores betweenthe images of the plurality of active focal planes.
 9. The method ofclaim 1, wherein: the images are scored such that a score is providedfor each region of interest visible in each image, the score beingindicative of the extent to which the respective region of interest isfocused in the image; the comparison of scores comprises determiningmathematical relationships for each of a plurality of active focalplanes that characterize the relationships between: the scores of theextent to which the on-axis region of interest is focused in the imagescaptured by the respective active focal plane and the spatialrelationship between the lens stack array and the sensor; and the scoresof the extent to which the off-axis region of interest is focused in theimages captured by the respective active focal plane and the spatialrelationship between the lens stack array and the sensor.
 10. The methodof claim 1, wherein the first predetermined criterion is based upon: ateach spatial relationship defined by the computed planes, averaging thescores indicative of the extent to which the on-axis region of interestis focused, the scores being averaged across all active focal planes atthe respective spatial relationship; and assessing the variance in theaverage scores between the spatial relationships.
 11. The method ofclaim 1, wherein the second predetermined criterion is based upon: ateach spatial relationship defined by the computed planes, averaging thescores indicative of the extent to which the off-axis region of interestis focused, the scores being averaged across all active focal planes atthe respective spatial relationship; and assessing the variance in theaverage scores between the spatial relationships.
 12. The method ofclaim 1 further comprising varying the spatial relationship between thelens stack array and the sensor.
 13. The method of claim 12, whereinvarying the spatial relationship between the lens stack array and thesensor involves sweeping the lens stack array relative to the sensor.14. The method of claim 13, wherein the lens stack array is swept in adirection substantially normal to the surface of the sensor.
 15. Themethod of claim 12, wherein varying the spatial relationship between thelens stack array and the sensor involves at least one of: tipping,tilting or rotating the lens stack array relative to the sensor.
 16. Themethod of claim 1, wherein a spatial relationship between the lens stackarray and the sensor is selected to be used when constructing an arraycamera module based upon the final spatial relationship arrangement. 17.The method of claim 1, wherein the resulting score provides anindication to which at least two regions of interest within the firstknown target are focused in the at least one image.
 18. The method ofclaim 17, wherein the at least two regions of interest include anon-axis region of interest and an off-axis region of interest.
 19. Themethod of claim 17, wherein the at least one image is scored such that ascore is provided for each region of interest visible in each image, theresulting score being indicative of the extent to which each respectiveregion of interest is focused in the image.